docs: qwen4exp results, an upstream-sync warning, and a caveat on "bit-identical" - #10
docs: qwen4exp results, an upstream-sync warning, and a caveat on "bit-identical"#10cpuchip wants to merge 2 commits into
Conversation
… caveat Ran the expert cache on Qwen3.8-Flash-Next (qwen4exp, 177B, 512 experts/layer) on a single RTX 3090: +34.5% decode for 9.2 GB, where moving whole layers with --n-cpu-moe instead spent 12.5 GB to buy 10%. Three things a downstream user would want in the guide: 1. After syncing past ggml-org ebb546b (ggml-org#26802), the pack chains trip GGML_ASSERT(ggml_cuda_mul_mat_id_needs_sync) on the first decode. op_params[0]=1 correctly disqualifies both fast paths, but the new predicate predates the flag and answers 'no sync needed' for a small quantized batch. Two-line fix included; not a code PR because this branch has no needs_sync to patch. 2. On qwen4exp the output is NOT bit-identical to baseline -- it diverges at temperature 0, consistent with float non-associativity between one fused chain and two summed pack chains. Quality is unaffected as far as a proper test can see: paired over 64 chunks, dPPL +0.047% (t=+0.88) against that test's own 0.107% detection floor. 3. --moe-cache-slots without --moe-cache-profile is SILENT. Added to the troubleshooting table: tok/s reads as a small regression and VRAM is the only tell.
github-actions
Bot
added
the
documentation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
The first draft cited only Pride and Prejudice, and that corpus is a poor discriminator: baseline PPL 1.0594 means the model is near-certain of every token, so there is little room for a defect to show. Re-ran the whole comparison on technical prose (NASA SP-4205 OCR, baseline PPL 8.2438). NASA cache-64 vs baseline -0.047% t -0.85 floor 0.112% Austen cache-64 vs baseline +0.113% t +1.68 floor 0.107% Neither detectable, and the SIGN FLIPS between corpora -- a real quality cost would keep its sign. Both rows shown rather than only the favourable one.
|
Updated the quality evidence — the first version cited only Pride and Prejudice, and that corpus turned out to be a poor discriminator: baseline PPL 1.0594 means the model is near-certain of every token, which we read as memorisation rather than a usable test. Re-ran the whole comparison on technical prose the model has not swallowed (NASA SP-4205 OCR, baseline PPL 8.2438):
64 paired chunks each, cached arm's peak VRAM 15,544 MiB against 6,286 for cache-off so it provably engaged. Neither delta is detectable — and the sign flips between corpora, which a real quality cost would not do. Both rows are in the diff rather than only the favourable one. So the claim I would stand behind for One near-miss worth passing on, since anyone comparing perplexity runs can hit it: mid-run I read the cached arm at chunk 38 (cumulative 9.29) against the other arms' final values (8.24) and it looked 12.7% worse. Cumulative perplexity is not comparable across different chunk counts. At matched indices the cached arm was the lowest of the three at every checkpoint. |
1 participant
Ran your expert cache on Qwen3.8-Flash-Next (
qwen4exp, 177B, 512 experts/layer, 10+1 active) on a single RTX 3090 — five cherry-picks onto ggml-org PR ggml-org#27742 and it worked. Three things a downstream reader of the guide would want, all docs-only.1. A collision waiting on your next upstream sync. ggml-org
ebb546b7e(ggml-org#26802, 2026-08-11) addsggml_cuda_mul_mat_id_needs_sync()and asserts it at the singlemul_mat_idcaller. Yourop_params[0] = 1correctly disqualifies both fast paths, so the op reaches the sync path — but the new predicate predates the flag and answers "no sync needed" for a small quantized batch. First decode, every time:Two lines fix it, on the branch linked in the README text. Not sent as a code PR because
perfdoesn't carryneeds_syncyet — there's nothing for the patch to apply to. Ready whenever you take the sync.2. A caveat on "bit-identical". On
qwen4expthe cached output is not bit-identical — it diverges from baseline at temperature 0, coherent and on-topic, consistent with float non-associativity between one fused chain and two summed pack chains. Quality is unaffected as far as a proper test can tell: paired over the same 64 chunks, ΔPPL +0.047% (t = +0.88), below that test's own 0.107% detection floor, with the cached arm's peak VRAM confirming it engaged. Suggesting statistically indistinguishable rather than bit-identical for architectures beyond the three you measured — your claim may well hold exactly on those.3. The silent no-op, added to your troubleshooting table:
--moe-cache-slotswithout--moe-cache-profiledoes nothing, reports clean, and reads as a small regression in tok/s. VRAM is the only tell — it doesn't move a megabyte. Cost us an evening.The numbers
--n-cpu-moe 48, no cache--n-cpu-moe 48+--moe-cache-slots 64+ profile--n-cpu-moe 40(whole layers on GPU instead)The comparison we found most useful: moving whole layers onto the card spent 12.5 GB to buy 10%, the cache spent 9.2 GB to buy 34.5%. Oracle hit rates from
llama-moe-trace+ yoursimulate.py: 68.2% at 64 slots/layer, 88.0% at 128 (which OOMs on 24 GB).We went in expecting the routing to be too flat for a cache to help — 512 fine-grained experts with load balancing look designed against one, and Laguna measured barely-skewed. It isn't: only 51.5% of (layer, expert) pairs are ever used.
Falsifier on all of the above: one card, one model, one quant (unsloth
UD-Q4_K_XL), routing profile traced from a single prompt. Happy to adjust or drop any of it.